Light source estimating device, light source estimating method, and imaging device and image processing method

ABSTRACT

A light source estimation method of this invention estimates from the sensor response the color characteristics of an unknown light source of an image-pickup scene, in order to improve white balance adjustment and other aspects of the quality of color reproduction; a projection conversion portion  6  projects sensor response values  5  into an image distribution  9  in an evaluation space not dependent on the image-pickup light source  2  using parameters obtained by operations which can be calorimetrically approximated from spectral sensitivity characteristics of image-pickup unit  4,  which are known, and from spectral characteristics of an assumed test light source  1;  an evaluation portion  10  evaluates the correctness of a plurality of the test light sources  1  based on the distribution state of sample values of the projected scene; and accordingly, the correct image-pickup light source  2  is estimated.

TECHNICAL FIELD

The present invention relates to a light source estimation apparatus,light source estimation method, image-pickup apparatus, and imageprocessing method in which, for example, image-pickup means having aplurality of different spectral sensitivity characteristics uses sensorresponse values obtained when photographing the image of an unspecifiedarbitrary object to estimate the spectral characteristics indicating thecolor of the unknown photographing light source which had beenirradiating an object.

BACKGROUND ART

The light which is incident on the human eye to enable vision is aportion of the radiant energy due to illumination which has beenreflected by an object which is seen and has propagated through the air;although the human vision system cannot directly measure thecharacteristics of objects and illumination, objects can be identifiedwith a degree of reliability even under illumination having unknownchromatic characteristics. This property is called color constancy, andfor example enables a white object surface to be perceived as white.

On the other hand, in digital still cameras, digital video cameras andother electronic image-pickup equipment, scenes are picked up as imagesthrough the response of a CCD (Charge Coupled Device) or otherphotosensor; however, because in general the balance of sensor responseamong the R, G, B, or other color channels is constant, in order to forman image in a state in which the appearance is natural in accordancewith the scene illumination, a correction mechanism is necessary toadjust the balance between channels. If the balance is not adequatelyadjusted, to the viewer of the image, places normally recognized asachromatic objects will be reproduced as colored in the image, orobjects will be reproduced with a color different from the colorremembered, so an unnatural impression is imparted; hence balanceadjustment is extremely important for color reproduction of an image.

Balance adjustment among channels can be performed by, for example,correction of achromatic colors called white balance in the gainadjustment of each channel; by correcting the color rendering propertiesof the light source through linear matrix transformation of signalsamong channels (Patent Reference 1); or by matching to the differentsensitivity responses of the sensors of image-pickup equipment, visionsystems and similar. However, whichever method is used, the correctionmechanism must use some means to obtain correction parametersappropriate to the scene. For example, the following equations (1) and(2) can be used to calculate appropriate gain values for adjustment ofthe white balance of sensors with a RGB three-channel response which islinear with respect to the quantity of light, together with the spectralsensitivity characteristics of the image-pickup system, if the spectraldistribution of the light source for the photographed scene is known.

$\begin{matrix}{\begin{bmatrix}R_{w} \\G_{w} \\B_{w}\end{bmatrix} = {SI}} & (1)\end{matrix}$

where S is a matrix indicating the sensor sensitivity (threechannels×number n of wavelength samples), and I is a column vectorindicating the spectral distribution of the light source (number n ofwavelength samples).g _(R) =G _(w) /R _(w) ,g _(G) =G _(w) /G _(w)=1.0,g _(B) =G _(w) /B_(w)   (2)

However, for the image-pickup equipment, information relating to objectsexisting in the scene at the time of image pickup without calibration orthe like and the illuminating light sources of the scene are normallyunknown; and adjustment parameters appropriate to the scene, or thechromatic characteristics of the illuminating light source necessary todetermine those parameters, must be identified from the response resultsof a dedicated sensor or sensor for image pickup, constituting a problemknown as the light source estimation problem or the color constancyproblem.

In the field of vision studies, various algorithms and calculationmodels began to be proposed from around 1980, and apart from these,techniques based on empirical knowledge have been incorporated inconventional color image-pickup equipment, the estimation performance ofwhich has advanced through the years. Recently, applications to roboticsand other artificial vision systems have also been anticipated.

One of the most widely used algorithms extracts the color components ofthe light source from average values of sensor response and theprojection thereof onto a black body locus, based on the assumption thatthe spatial average over the scene of the surface reflectivity of anobject is close to gray (Non-patent Reference 1, Non-patent Reference2), and is used in a variety of modes, such as simply averaging thesensor response among pixels, averaging pixels within the range of aspecified brightness level, or changing the range or weighting ofsampling depending on the position in space. There are also a method inwhich color components of the light source are extracted from samplingresults for pixels with high response values, assuming that the areawith the highest brightness level corresponds to a white surface closeto a perfectly diffuse reflecting surface (Patent Reference 2), and amethod in which an area of high brightness level is assumed to be aspecular reflecting component, and the light source is estimated fromthe distribution of the response values (Non-patent Reference 3).Because these methods are based on an assumption about an objectsurface, which should be physically independent of the light source, itis known that depending on the scene, the results of light sourceestimation may be greatly affected by the state of an object whichdeviates from the assumptions made.

There are also a study in which, by assuming a reflection model in whichan object surface is a diffuse reflecting surface, and approximating thespectral characteristics of the light source and of the object surfaceby a linear model of few dimensions, reconstruction is attempted throughlinear calculations using a vector space different from that of thesensor response (Non-patent Reference 4), and a study in whichconstraining conditions, such as that the spectral reflectivity of anobject surface must physically be in the range 0 to 1, are applied toselect a light source with high probability (Non-patent Reference 5);however, in generalized image-pickup systems with few response channels,these do not independently provide sufficient estimation performance.Further, although the volume of computations is increased, there hasalso been proposed a method of integrating a plurality of knownassumptions and probabilistic distributions for the light source, objectsurfaces, image-pickup system and similar, to improve the accuracy ofstatistical estimation (Non-patent Reference 6).

In methods which apply reflection models in particular, rather thanperforming an estimate taking as the solution a single completelyunknown light source, in some methods wide prior knowledge is utilizedin a method of determination in which the most probable light sourcesare categorized or detected from among a number of light sourcesselected in advance as candidates; such methods may be advantageous inthat calculations are comparatively simple and results can be outputrapidly. As criteria for judging the reliability of the result, errorsby restoring the sensor response itself under a fixed constraintcondition may be used (Non-patent Reference 7) ; and there have beenproposals for widely using distribution states in the color gamut withinthe sensor space, to efficiently quantify a correlation relationshipthrough comparison with a color gamut, adopted in advance as areference, or a weighted distribution (Non-patent Reference 8,Non-patent Reference 9, Non-patent Reference 10, Patent Reference 3).

Patent Reference 1: Published Japanese Patent Application No.2002-142231

Patent Reference 2: Published Japanese Patent Application No. H9-55948

Patent Reference 3: Published Japanese Patent Application No. H5-191826

Non-patent Reference 1: G. Buchsbaum, “A Spatial Processor Model forObject Color Perception”, J. Franklin Inst., 310, 1980

Non-patent Reference 2: E. H. Land, “Recent Advances in Retinex Theory”,Vision Research, 26, 1986

Non-patent Reference 3: H. C. Lee, “Method for computing thescene-illuminant chromaticity from specular highlights”, J. Opt. Soc.Am. A, Vol. 3, No. 10, 1986

Non-patent Reference 4: L. T. Maloney & B. A. Wandell, “Color Constancy:A method for recovering surface spectral reflectance”, J. Opt. Soc. Am.A, 1986

Non-patent Reference 5: D. A. Forsyth, “A Novel Algorithm for ColorConstancy”, Int. J. Comput. Vision, 5, 1990

Non-patent Reference 6: D. H. Brainard & W. T. Freeman, “Bayesian colorconstancy”, J. Opt. Soc. Am. A, Vol. 14, No. 7, 1997

Non-patent Reference 7: B. Tao, I. Tastl & N. Katoh, “IlluminationDetection in Linear Space”, Proc. 8th Color Imaging Conf., 2000

Non-patent Reference 8: Hewlett-Packard Company, Hubel et al., “Whitepoint determination using correlation matrix memory”, U.S. Pat. No.6,038,339

Non-patent Reference 9: G. D. Finlayson, P. M. Hubel & S. Hordley,“Color by correlation”, Proc. 5th Color Imaging Conf., 1997

Non-patent Reference 10: S. Tominaga & B. A. Wandell, “Naturalscene-illuminant estimation using the sensor correlation”, Proc. IEEE,Vol. 90, No. 1, 2002

DISCLOSURE OF THE INVENTION

Generally, in order to use a light source estimation algorithm in imageprocessing operations such as white balance processing within a digitalcamera, not only must the processing speed be fast, but at the time ofimplementation it is necessary that the costs of memory consumption andsimilar be low.

However, even in the cases of those among the above-describedconventional algorithms which enable comparatively fast categorizationand detection (Non-patent Reference 8, Non-patent Reference 9, PatentReference 3), as indicated by the conceptual diagram of a conventionalmethod to evaluate the reasonableness of test light sources in sensorresponse space in FIG. 10, while there is the possibility of improvingthe estimation accuracy as the number 1 to n of test light sources 101set as candidates is increased, because comparative evaluations of theimage distribution 106 for sensor response 105 to a picked-up image ofan object 103 picked up by the image-pickup means 104 with referencedistributions (1, 2, . . . , n) 108 stored in storage media 107corresponding to the respective test light sources 101, are performed,by the comparison portion 109, in an image distribution 106 in sensorspace dependent on the image-pickup light source 102, with scores ofevaluation results (1, 2, . . . , n) 110 output, and the test lightsource judged to be most correct based on the score values 110 is judgedto be the estimated light source O by the judgment portion 111;information for reference distributions (1, 2, . . . , n) 108 such asthe color gamut, weighted distribution, target values, and so on used ascomparison criteria for the correct light source must be held in storagemedia 107 in the same quantity as the number 1 to n of test lightsources 101 selected for use, so there is a tendency for increased useof the ROM or other storage media 107, and therefore the problem of acombination of advantages and disadvantages with respect to accuracy andcost has remained.

Whereas the above methods create one image distribution for one pickedup image by a fixed projection, and make the distribution becomejudgment criteria as compared with a plurality of referencedistributions corresponding to a plurality of light sources assumed, inthe present invention, a plurality of reference distributions generatedby projections corresponding to a plurality of assumed light sources arecompared with a single fixed reference distribution to be employed asthe judgment criterion.

The present invention was devised in light of the above, and has as anobject to provide a light source estimation apparatus, light sourceestimation method, image-pickup apparatus, and image processing methodfor estimating the color characteristics of an unknown light source ofan image-pickup scene from the sensor response, in order to improve theautomatic white balance adjustment and other aspects of colorreproduction quality of color image-pickup equipment.

A light source estimation apparatus of this invention estimates thecorrect image-pickup light source by including: storage means forstoring, for each test light source, parameters for projecting sensorresponse values into an evaluation space not dependent on theimage-pickup light source by performing, for the sensor response values,operations which can be calorimetrically approximated from a pluralityof different known spectral sensitivity characteristics of image-pickupmeans and spectral characteristics of a plurality of test light sourcesassumed in advance; projection conversion means for projecting thesensor response values into the evaluation space not dependent on theimage-pickup light source using parameters stored in the storage means;and evaluation means for evaluating the correctness of a plurality oftest light sources based on the image distribution state of samplevalues of an image scene projected by the projection conversion means.

Hence according to this invention, the following action is achieved.

With respect to a sampled sensor response, through operations which canbe calorimetrically approximated from the known spectral sensitivitycharacteristics of the image-pickup system and from spectralcharacteristics of test light sources, projection into an evaluationspace not dependent on the light source is performed, and thereasonableness of each of the test light sources is evaluated based onthe states of sample values widely distributed therein.

Accordingly, it is necessary only to store, with respect to each testlight source, a matrix or other parameters for projection from thesensor space into the evaluation space, so that by providing evaluationcriteria in a single evaluation space, high estimation accuracy can beobtained with low memory consumption.

Further, a light source estimation method of this invention correctlyestimates, for sensor response values, the image-pickup light source, byperforming projection into an evaluation space not dependent on theimage-pickup light source through operations which can becalorimetrically approximated from known spectral sensitivitycharacteristics of image-pickup means and from spectral characteristicsof assumed test light sources, and by evaluating the correctness of aplurality of test light sources based on the state of distribution ofsampled values of the projected scene.

Hence according to this invention, the following action is achieved.

In order to perform evaluations using a fixed space not dependent on thelight source, it is sufficient to hold information, as comparisoncriteria for the correct light source, only for a single referencedistribution space, so that evaluation processing is simplified, andconsequently the problem of increasing costs can be resolved. As afurther consequence, a greater amount of information (conditions anddata) for referencing as criteria for the correct light source can beprovided, so that optimization adjustment to improve estimation accuracyis also facilitated.

Further, an image-pickup apparatus of this invention includes: storagemeans for storing, for each test light source, parameters for projectingsensor response values into an evaluation space not dependent on theimage-pickup light source by performing, for the sensor response values,operations which can be calorimetrically approximated from a pluralityof different known spectral sensitivity characteristics of image-pickupmeans and spectral characteristics of a plurality of test light sourcesassumed in advance; projection conversion means for projecting sensorresponse values into the evaluation space not dependent on theimage-pickup light source using parameters stored in the storage means;evaluation means for estimating the correct image-pickup light source byevaluating the correctness of a plurality of test light sources based onthe image distribution state of sample values of an image sceneprojected by the projection conversion means; light source estimationmeans for estimating the final image-pickup light source to bedetermined as the estimated light source by conjoining in numericalformulas, or by selecting through conditional branching, or by combiningboth of an image-pickup light source determined by estimation and alight source determined by an estimation method different from theestimation method used; and color balance adjustment means which usesspectral characteristics or parameters appropriate thereto, as the colorof the estimated image-pickup light source, in color balance processingof the sensor response of the image-pickup means.

Hence according to this invention, the following action is achieved.

In this image-pickup apparatus, the range of estimation of theimage-pickup light source can be broadened, and by storing, for eachtest light source, only a matrix or other parameters for projection fromthe sensor space into the evaluation space and by providing evaluationcriteria in a single evaluation space, high estimation accuracy with lowmemory consumption is obtained to be used in color balance processing.

Further, an image processing method of this invention performsprojection, for sensor response values, into an evaluation space notdependent on the image-pickup light source through operations which canbe calorimetrically approximated from known spectral sensitivitycharacteristics of image-pickup means and from spectral characteristicsof assumed test light sources; estimates the correct image-pickup lightsource by evaluating the correctness of a plurality of test lightsources based on the distribution state of sample values of theprojected scene; estimates the final image-pickup light source to bedetermined as the estimated light source by conjoining in numericalformulas, by selecting through conditional branching, or by combiningboth of, an image-pickup light source determined by estimation and alight source determined by an estimation method different from theestimation method used; and uses the spectral characteristics orparameters appropriate thereto, as the color of the estimatedimage-pickup light source, in color balance processing of the sensorresponse of the image-pickup means.

Hence according to this invention, the following action is achieved.

In this image processing method, the range of estimation of theimage-pickup light source can be broadened, and by storing, for eachtest light source, only a matrix or other parameters for projection fromthe sensor space to the evaluation space, and by providing evaluationcriteria in a single evaluation space, high estimation accuracy withfast processing is obtained to be used in color balance processing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram of a method of evaluating thereasonableness of test light sources in an evaluation space notdependent on the light source, which is applied to an embodiment of thepresent invention;

FIG. 2 is an image processing block diagram within a digital stillcamera;

FIG. 3 is a flowchart showing reference distribution generationprocessing;

FIG. 4 is a flowchart showing light source estimation processing;

FIG. 5 is a diagram showing a spectral reflectivity basis function;

FIG. 6 is a diagram showing an example of projecting a color chart intoa reflectivity vector space;

FIGS. 7A to 7C are diagrams showing test light sources, in which FIG. 7Ais at equal intervals, FIG. 7B is finely divided at specific intervals,and FIG. 7C includes a plurality of light sources;

FIG. 8 is a diagram showing a distribution of reflectivity samples;

FIG. 9 is a diagram showing a reference distribution table; and,

FIG. 10 is a conceptual diagram of a conventional method of evaluatingthe reasonableness of test light sources in sensor response space.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention are explained,referring to the drawings as appropriate.

FIG. 1 is a conceptual diagram of the method, which is applied in thisembodiment, of evaluating the reasonableness of test light sources in anevaluation space not dependent on the light source.

In FIG. 1, an image of an object 3 is picked up by image-pickup means 4using an image-pickup light source 2. In order to enable projection intoan evaluation space not dependent on the light source, with respect to asensor response 5 sampled by the image-pickup means 4 at this time,calorimetric approximation operations are performed in advance usingspectral sensitivity characteristic of the image-pickup means 4, whichis a known quantity, and using spectral characteristics of test lightsources (1 to n) 1, and a matrix (1 to n) 8 corresponding to each of thetest light sources, which is stored in storage media 7, is used toperform projection into image distribution 9 of the evaluation space bymeans of a projection conversion portion 6; then based on the states ofsample values distributed widely in the image distribution 9, anevaluation score value 12 is output by an evaluation portion 10 withrespect to the reasonableness of each test light source (1 to n) 1 basedon a reference distribution 11, and a judgment portion 13 then judgesthe test light source, which is judged to be most nearly correct basedon the score values 12, to be the estimated light source O.

Accordingly, it is sufficient to store in the storage media 7 only amatrix (1 to n) 8 or other parameters for projection from the sensorspace to the image distribution 9 in the evaluation space with respectto each of the test light sources (1 to n), so that by providingevaluation criteria through a single reference distribution 11 in theimage distribution 9 in the evaluation space, scores (1, 2, . . . , n)12 having a sufficient amount of information to enable accurate judgmentby the judgment portion 13 can be output by the evaluation portion 10,with only a small amount of storage space of the storage media 7 used.

FIG. 2 is a block diagram showing an image processing system within adigital still camera, to which an embodiment of this invention isapplied.

A digital still camera is assumed, in which sensor response due to thespectral sensitivity characteristics, differing with respect to eachpixel, in the three channels red, blue, green can be obtained as 10-bitdigital values proportional to the quantity of light, and in the imageprocessing operation unit within the apparatus, processing for whitebalance adjustment is performed using appropriate gain values for eachchannel.

In order to determine appropriate gain values for white balanceadjustment based on sensor response obtained by subtracting offsetcomponents in a black correction portion 22 from values read by a sensorresponse readout portion 21, subsampling is performed at appropriateposition intervals among all pixels by a subsampling portion 23. At thistime, pixels which can be judged to have reached saturation in rangesnear the minimum and maximum values of the sensor response, areexcluded. Light source estimation processing described later on isperformed by a light source estimation processing portion 24 on thosesampled pixels.

As a result, a gain value corresponding to the estimated light source isselected from among the gain values for white balance adjustment foreach test light source stored in a gain determination portion 25 throughthe calculations indicated previously in equations (1) and (2); and theselected gain value is employed in white balance processing in a whitebalance gain adjustment portion 26.

Subsequently, through nonlinear grayscale conversion by a grayscaleconversion portion 27 and 3×3 matrix conversion by a color spaceconversion portion 28, Y, Cb and Cr (luminance/color-difference signals)are converted into 8 bits each, encoding by an encoding portion 29 whichincludes image compression processing is performed, and the result iswritten by a file writing portion 30 as an electronic file in a memorycard.

In light source estimation in this embodiment, an object surface linearreflection model is assumed as indicated by the following equation (3)with respect to the sensor response.

$\begin{matrix}{f = {\begin{bmatrix}R \\G \\B\end{bmatrix} = {SLr}}} & (3)\end{matrix}$

where L is the diagonal matrix (n×n) containing n wavelength samples oflight source spectral distributions, and r is the column vector (numberof wavelength samples n) indicating the object surface spectralreflectivity.

This embodiment is explained using the matrix calculation of equation(4), which assumes that the spectral reflectivity of the object surfacecan be approximated by a linear combination of three basis functions.

$\begin{matrix}{{r \cong r_{a}} = {B_{w} = {\left\lbrack {b_{1}\mspace{20mu} b_{2}\mspace{20mu} b_{3}} \right\rbrack\begin{bmatrix}\beta_{1} \\\beta_{2} \\\beta_{3}\end{bmatrix}}}} & (4)\end{matrix}$

where B is a matrix indicating the basis functions of the spectralreflectivity (number of wavelength samples n×basis number 3), b₁, b₂, b₃are column vectors indicating the basis functions of the spectralreflectivity (number of wavelength samples n), w is a column vectorcontaining weighting coefficients (basis number 3), β₁, β₂, β₃ areweighting coefficients used to indicate the spectral reflectivity aslinear sums of the basis functions, and r_(a) is a column vector (numberof wavelength samples n) indicating approximate values of the spectralreflectivity.

If the spectral reflectivity is known, approximate values for weightingcoefficients of the basis functions can be calculated as in thefollowing equation (5).

$\begin{matrix}{w = {\begin{bmatrix}\beta_{1} \\\beta_{2} \\\beta_{3}\end{bmatrix} = {{B^{t}\left( {BB}^{t} \right)}^{- 1}r}}} & (5)\end{matrix}$

Because in equation (5) w does not depend on the sensor or theimage-pickup light source, the vector space resulting from the weightingcoefficients of basis functions (hereafter called the reflectivityvector space) can be said to be a space specific to the object. Thespectral reflectivity basis functions shown in FIG. 5 are examples ofbasis functions shown in the wavelength range from 400 nm to 700 nm;because β1 represents the brightness component, the first component isset to be flat over the wavelength range, whereas the second and thirdcomponents are the results of extracting the highest two componentsexcluding the first component as an offset from the spectralreflectivity data for 24 specific color charts, and then performingprincipal component analysis. From equations (3) and (4), when the basisnumber and the number of sensor response channels are the same, thecolumn vector projected from the sensor response by the matrix iscalculated using the following equation (6).

$\begin{matrix}{\;{\overset{\sim}{w} = {\begin{bmatrix}{\;{\overset{\sim}{\beta}}_{1}} \\{\;{\overset{\sim}{\beta}}_{2}} \\{\;{\overset{\sim}{\beta}}_{3}}\end{bmatrix} = {{({SLB})^{- 1}\begin{bmatrix}R \\G \\B\end{bmatrix}} = {M\begin{bmatrix}R \\G \\B\end{bmatrix}}}}}} & (6)\end{matrix}$

The matrix M in equation (6) projects the sensor response intoreflectivity vector space, but is a matrix which depends on the lightsource L, and is called hereinafter the light source matrix. If the samelight source as that for the scene for which the sensor response wasobtained is used as the light source L, the column vector {tilde over(w)} projected from the sensor response is restored to a close valueeven if the spectral reflectivity of the object is unknown. However, ifa light source different from that for the image-pickup scene is used,this restoration accuracy is not obtained. Hence an arbitrary lightsource Li is assumed, and the light source matrix M_(i) indicated inequation (7) is used.M _(i)=(SL _(i) B)⁻¹   (7)

Using the light source matrix M_(i) obtained using equation (7), andutilizing the relation between the column vector {tilde over (w)}_(i)projected as equation (6) and the correct column vector w for theobject, the degree of similarity with the image-pickup light source inthe reflectivity vector space can be evaluated. FIG. 6 shows, bysimulation results for 24 color charts in the β₂-β₃ plane in thereflectivity vector space, that the distribution of points approximatedfrom the spectral reflectivity of a known surface is in a state close tothe distribution of points projected using the light source matrix forthe same light source as that at the time of image pick up from thesensor response for a pickup image of the same surface, and that thedistribution of points projected using a light source matrix for a lightsource different from that at the time of image pickup is in a statedifferent from the former distribution.

The column vectors w which can be adopted by the object are widelydistributed in reflectivity vector space, and it is difficult toevaluate the relation with an unknown object by using the column vector{tilde over (w)}_(i) obtained from the sensor response for a singlepixel. Hence here it is assumed that the image-pickup scene isilluminated uniformly by a single light source, the sensor response forsampled pixels among all pixels is projected into the reflectivityvector space, and by evaluating these distribution states (hereaftercalled the image distribution), a single estimated light source isdetermined. A plurality of light sources for evaluation (hereaftercalled test light sources) are provided, and all light source matrixesare calculated and stored in advance according to the above equation(7).

At the time of estimation processing, each light source matrix isemployed with respect to all test light sources to evaluate theprojected image distribution, and the test light source with the highestevaluation index indicating the degree of correctness is selected as theestimated light source among all the test light sources. Here, naturallight with color temperatures in the range approximately 2850K to 8000Kare assumed as test light sources; in order to reduce estimation errorscattering, seven test light sources were adopted on the CIE daylightlocus in the u′-v′ plane of the CIE 1976 UCS chromaticity diagram atintervals as nearly equal as possible, as shown in FIG. 7A. In thisembodiment, any of the test light sources can be selected as theestimation result, so that as shown in FIG. 7B, the test light sourcesare more finely divided only in specific intended intervals in the colortemperature direction on the u′-v′ plane; and as shown in FIG. 7C, lightsources employing physically different light emission methods, such asfluorescent lamps, may also be added to raise the probability ofobtaining the correct estimation result in more diverse scenes.

In order to evaluate in relative terms whether the distribution state ofan image distribution is the correct state for an object, a single fixeddistribution state (hereinafter called the reference distribution) to betaken as the criterion for comparison in reflectivity vector space isreferenced. The reference distribution is stored as data in the formatof a two-dimensional numerical table provided with weightingcoefficients with respect to each cell divided at equal intervals in theβ₂-β₃ plane. This reference distribution is generated using, forexample, a procedure such as follows.

Specific reference distribution generation processing is shown in theflowchart of FIG. 3.

In step S1, spectral reflectivity data for the surfaces of numerousobjects which might be objects for image pickup is collected, andrepresentative samples which are as diverse as possible are extracted.

In step S2, sample data for spectral reflectivity is projected intoreflectivity vector space using equation (5) (FIG. 8).

In step S3, the cell region is set by specifying lower ends low₂ andlow₃, upper ends high₂ and high₃, and cell division numbers bin₂ andbin₃ for each axis of the β₂-β₃ plane in reflectivity vector space, suchthat the rectangular region comprehends the sample distribution.

In step S4, the numbers of data samples positioned in each cell regionare counted to generate a frequency distribution.

Cell coordinates (x, y) are calculated using the following equation (8).x=floor((β₂−low₂)×bin₂/(high₂−low₂))y=floor((β₃−low₃)×bin₃/(high₃−low₃))   (8)

where floor ( ) is an operator which discards the decimal fraction.

In step S5, values Tr_(xy) which encode frequencies in each cell to anappropriate bit depth are recorded.

In step S6, in order to form the outline of the distribution range ofthe reference distribution, the polygon which convexly encompasses thecells in which values exist, is calculated, and by assigning a value of1 to any cells existing within the polygon for which a value does notexist, holes in cells within the outline are filled. FIG. 9 showsnumerical values in cells in an example of a reference distributiongenerated with a bit depth of 2.

FIG. 4 is a flowchart of specific light source estimation processing.

At the time of estimation processing, calculation of score values withrespect to each test light source is repeated according to the followingprocedure shown in FIG. 4.

In step S11, a projection matrix is selected for a test light source.Specifically, the projection conversion portion 6 shown in FIG. 1selects a light source matrix M_(i) for a test light source i from thestorage media 7.

In step S12, reading of sample pixels is performed. Specifically, theprojection conversion portion 6 shown in FIG. 1 reads sample pixels fromthe image-pickup means 4. Here, the sensor response 5 of sample pixelsis the image-pickup result for various scenes.

In step S13, matrix conversion is performed. Specifically, theprojection conversion portion 6 shown in FIG. 1 uses the light sourcematrix M_(i) for the test light source i to project the sensor response5 for sample pixels into the reflectivity vector space.

In step S14, the image distribution is generated. Specifically, theprojection conversion portion 6 shown in FIG. 1 creates an imagedistribution 9 in the same cell positions as the reference distribution11. Similarly to generation of the reference distribution 11, the imagedistribution 9 includes values Th_(ixy) which encode the frequency ineach cell to an appropriate bit depth. Here the bit depth is taken to be1; an example of an image distribution in which a value of 1 is assignedto each cell in which one or more pixel exists, and 0 is used for allother cells, is shown in gray within the cells of the referencedistribution table in FIG. 9.

In step S15, processing is repeated for all sample pixels, returning tostep S12 to repeat the processing and judgment of steps S12 through S15.Specifically, the projection conversion portion 6 of FIG. 1 not onlyrecords the image distribution 9 for each pixel, but also counts pixelspositioned in cells (shown as the bold border in FIG. 9) for whichvalues exist in the reference distribution 11.

In step S16, score values are calculated with respect to each test lightsource. Here, a score value 12 is a correlation value or similar betweenthe image distribution 9 and the reference distribution 11.Specifically, the evaluation portion 10 shown in FIG. 1 calculates thefollowing three types of indexes.

First, as the distribution correlation, equation (9) is used tocalculate the sum of the product of the image distribution 9 withweightings of the reference distribution 11 for each cell, as an indexrepresenting the correlation function between the reference distribution11 and the image distribution 9.

$\begin{matrix}{{Ic}_{i} = {\sum\limits_{x = 1}^{{bin}\; 2}{\sum\limits_{y = 1}^{{bin}\; 3}{{Tr}_{xy}{Th}_{ixy}}}}} & (9)\end{matrix}$

Second, as the division of the number of pixels, equation (10) is usedto calculate the fraction of the number of pixels among all samplepixels existing in the color gamut (shown as the bold border in FIG. 9)of the reference distribution 11, as a comparative index relative to thereference distribution 11 with respect to the number of pixel samples.Ip _(i)=(number of pixels positioned at cell coordinates x,y for whichTr _(ixy)>0)/(total number of sample pixels)   (10)

Third, as the distribution size, equation (11) is used to calculate thefollowing index, only with respect to the image distribution inreflectivity vector space, based on the assumption that in a case ofprojection using an erroneous light source matrix, the larger thedifference with the correct light source matrix, the broader will be thedistribution range dispersed in the β₂ axis direction.Ir _(i)=(Max2_(m)−Min2_(m))/(Max2_(i)−Min2_(i))   (11)

where Max2 _(i) is the maximum value of β₂ projected by the light sourcematrix of the test light source i, Min2 _(i) is the minimum value of β₂projected by the light source matrix of the test light source i, and mis the light source i among all test light sources for which Max2_(i)-Min2 _(i) is smallest.

Here, the evaluation portion 10 shown in FIG. 1 uses equation (12) toobtain score values 12 for light sources i by multiplying the threetypes of index.S _(i) =Ic _(i) ·Ip _(i) ·Ir _(i)   (12)

In step S17, processing is repeated with respect to all test lightsources by returning to step S11 to repeat the processing and judgmentof steps S11 through S17.

In step S18, the estimated light source is selected. Specifically, theevaluation portion 10 shown in FIG. 1 determines as the estimated lightsource the light source i having the highest score value 12 uponobtaining score values 12 for all test light sources 1.

In addition, an intermediate light source, selected through weightedaveraging process etc. using test light sources with high score values,may also be determined to be the estimated light source.

In addition, only a specific interval close to test light sources withhigh score values on the u′-v′ plane in FIG. 7A may be more finelydivided in the color temperature direction to newly generate a pluralityof test light sources, and score calculation and decisions thereof areperformed in stages with respect to the newly generated test lightsources to further improve the resolution of the estimation result.

When test light sources include a plurality of light sources which canbe classified in different categories on the basis of the physicalmethod of light emission, such as for example high-efficiencyfluorescent lamps or three-wavelength fluorescent lamps, differentindexes between evaluations within each category and evaluations amongcategories, are used in respective calculations, and score valuesseparately obtained may be combined to judge the estimated light source.

When scene light source estimation processing is performed continuouslyover a period of time, indexes and estimation results obtained in thepast at short intervals may be combined to judge the most recentestimated light source.

In order to evaluate the correctness of a test light source inreflectivity vector space, in addition to the distribution state in theβ₂-β₃ plane, distributions in other two-dimensional spaces such as theβ₁-β₃ and β₁-β₂ planes may be evaluated, or evaluations ofone-dimensional distributions along each axis may be performed, or thedistribution state in three dimensions may be evaluated.

For example, by using a two-dimensional space based on relative valuesamong vector channels, such as for example the β₂/β₁-β₃/β₁ planeresulting from division of both β₂ and β₃ by β₁, the effects ofscattering in exposure for each scene and unevenness in the lightingintensity within the same scene on evaluations can be suppressed.

Further, sensor response values, which are the results of image pickupof various scenes actually existing or the results of prediction bynumerical operations of image pickup of diverse virtual scenes, can beprojected into evaluation spaces for separate scenes through operationswhich can be calorimetrically approximated from the spectral sensitivitycharacteristics of the image-pickup means and the spectral distributioncharacteristics of the image-pickup light source measured at the time ofimage pickup of each scene, and weighting distribution and regioninformation generated from the frequency distribution of the projectionvalues may be used as a reference color gamut.

Needless to say, this invention is not limited to the above-describedembodiment, but can adopt various other configurations as appropriatewithin the scope of the claims of this invention.

A light source estimation apparatus of this invention estimates thecorrect image-pickup light source by including: storage means forstoring, for each test light source, parameters for projecting sensorresponse values into an evaluation space not dependent on theimage-pickup light source by performing, for the sensor response values,operations which can be calorimetrically approximated from a pluralityof different known spectral sensitivity characteristics of theimage-pickup means and the spectral characteristics of a plurality oftest light sources assumed in advance; projection conversion means forprojecting sensor response values into the evaluation space notdependent on the image-pickup light source using parameters stored inthe storage means; and evaluation means for evaluating the correctnessof a plurality of test light sources based on the image distributionstate of sample values of an image scene projected by the projectionconversion means. Hence by projecting sampled sensor responses into anevaluation space not dependent on the light source through operationswhich can be calorimetrically approximated from the spectral sensitivitycharacteristics of the image-pickup system, which are known, and fromthe spectral characteristics of the test light source, there is theadvantageous result that the reasonableness of each test light sourcecan be evaluated based on the state of sample values widely distributedtherein.

Accordingly, it is sufficient to store with respect to each test lightsource only a matrix or other parameters necessary for projection fromthe sensor space into the evaluation space, so that by providingevaluation criteria in a single evaluation space, high estimationaccuracy can be obtained with low memory consumption.

Further, a light source estimation method of this invention correctlyestimates the image-pickup light source by performing projection ofsensor response values into an evaluation space not dependent on theimage-pickup light source through operations which can becalorimetrically approximated from spectral sensitivity characteristicsof the image-pickup means which are known and from the spectralcharacteristics of the assumed test light source, and by evaluating thecorrectness of a plurality of test light sources based on the state ofdistribution of sampled values of the projected scene. Hence evaluationis performed using a fixed space not dependent on the light source, sothat it is sufficient to store information for only one referencedistribution space as the comparison criterion for the correct lightsource, and evaluation processing is simplified, so that the problem ofcost increases can be resolved. Further, a greater amount of information(conditions and data) for referencing as criteria for the correct lightsource can be provided, and so there is the advantageous result thatoptimization adjustment to improve estimation accuracy is alsofacilitated.

In the method according to this invention, the most appropriate lightsource is judged from among a plurality of test light sources; inmethods proposed in the prior art, an evaluation criterion is necessaryfor each light source in order to perform evaluations in a space whichdepends on the light source, and because the amount of data used asevaluation criteria increases in proportion to the number of patterns ofthe test light sources, either the amount of data of evaluation criteriaor the number of test light sources must be reduced, so estimationaccuracy is sacrificed, or accuracy is given priority, resulting inincreased memory costs. In this invention, coefficients for spaceconversion are provided which require only a small amount of memory foreach test light source, and evaluations are performed using a fixedspace not dependent on the light source, so that it is sufficient tostore with respect to only a single space the information (conditionsand data) to be referenced as comparison criteria for judging thecorrect light source, and consequently the estimation accuracy can beimproved without increases in cost, affording advantages over thetechniques of the prior art.

Further, an image-pickup apparatus of this invention includes: storagemeans for storing, for each test light source, parameters for projectingsensor response values into an evaluation space not dependent on theimage-pickup light source by performing operations which can becalorimetrically approximated, for the sensor response values, from aplurality of different known spectral sensitivity characteristics of theimage-pickup means and the spectral characteristics of a plurality oftest light sources assumed in advance; projection conversion means forprojecting sensor response values into the evaluation space notdependent on the image-pickup light source using parameters stored inthe storage means; evaluation means for estimating the correctimage-pickup light source by evaluating the correctness of a pluralityof test light sources based on the image distribution state of samplevalues of an image scene projected by the projection conversion means;light source estimation means for estimating the final image-pickuplight source to be determined as the estimated light source byconjoining in numerical formulas, or by selecting through conditionalbranching, or by combining both of, an image-pickup light sourcedetermined by estimation and a light source determined by an estimationmethod different from the estimation method used; and color balanceadjustment means for using spectral characteristics, which are the colorof the estimated image-pickup light source, or parameters appropriatethereto in color balance processing of the sensor response of theimage-pickup means. Hence in the image-pickup apparatus, the range ofestimation of the image-pickup light source can be broadened, and it isonly necessary to store a matrix or other parameters for each test lightsource for projection from sensor space into evaluation space, and sothere is the advantageous result that by providing evaluation criteriain a single evaluation space, high estimation accuracy can be obtainedwith low memory consumption, enabling use in color balance processing.

Further, an image processing method of this invention performsprojection, for sensor response values, into an evaluation space notdependent on the image-pickup light source through operations which canbe calorimetrically approximated from known spectral sensitivitycharacteristics of the image-pickup means and from the spectralcharacteristics of the assumed test light source; estimates the correctimage-pickup light source by evaluating the correctness of a pluralityof test light sources based on the distribution state of sample valuesof the projected scene; estimates the final image-pickup light source tobe determined as the estimated light source by conjoining in numericalformulas, or by selecting through conditional branching, or by combiningboth of, an image-pickup light source determined by estimation and alight source determined by an estimation method different from theestimation method used; and uses the spectral characteristics, which arethe color of the estimated image-pickup light source, or parametersappropriate thereto in color balance processing of the sensor responseof the image-pickup means. Hence in the image processing method, therange of estimation of the image-pickup light source can be broadened,and it is only necessary to store a matrix or other parameters for eachtest light source for projection from sensor space into evaluationspace, and so there is the advantageous result that by providingevaluation criteria in a single evaluation space, high estimationaccuracy can be obtained with low memory consumption, enabling use incolor balance processing.

This invention can provide one framework for accurately estimating thelight source of an image-pickup scene from the response of theimage-pickup system. If the light source of an unknown scene can beestimated in the image-pickup system, it becomes possible to accuratelydetermine the image white balance adjustment, color matching adjustmentand other parameters in the image-pickup equipment, and accurate colorreproduction of the picked up image of a scene, and accurate correctionso as to obtain a specific intended color reproduction, can be performedwhen recording and displaying images.

1. A light source estimation apparatus to correctly estimate theimage-pickup light source, in which from sensor response values obtainedupon pickup of an image of an unspecified arbitrary object, image-pickupmeans having a plurality of different spectral sensitivitycharacteristics estimates spectral characteristics indicating color ofan unknown image-pickup light source irradiating an object, comprising:storage means for storing, for each test light source, parameters forprojecting said sensor response values into an evaluation space notdependent on said image-pickup light source by performing operationswhich can be colorimetrically approximated from a plurality of differentsaid known spectral sensitivity characteristics of said image-pickupmeans and from the spectral characteristics of a plurality of test lightsources assumed in advance; projection conversion means for projectingsaid sensor response values into said evaluation space not dependent onthe image-pickup light source using parameters stored in said storagemeans; and, evaluation means for evaluating the correctness of saidplurality of test light sources based on the image distribution state ofsample values of an image scene projected by said projection conversionmeans.
 2. A light source estimation method in which from sensor responsevalues obtained upon pickup of an image of an unspecified arbitraryobject, image-pickup means having a plurality of different spectralsensitivity characteristics estimates spectral characteristicsindicating color of an unknown image-pickup light source irradiating anobject, comprising the steps of: projecting said sensor response valuesinto an evaluation space not dependent on the image-pickup light sourcethrough operations which can be calorimetrically approximated from knownspectral sensitivity characteristics of image-pickup means and fromspectral characteristics of an assumed test light source; and estimatingthe correct image-pickup light source by evaluating the correctness of aplurality of said test light sources based on a state of distribution ofsampled values of the projected scene.
 3. The light source estimationmethod according to claim 2, wherein a vector space for said evaluationis a space in which weighting coefficients used to approximate thespectral reflectivity of diverse object surfaces by conjoining aplurality of reflectivity basis functions, represent the spectralreflectivity characteristics specific to an object surface, or a spacein which the weighting coefficients become further converted valuesthrough fixed operations.
 4. The light source estimation methodaccording to claim 3, wherein said reflectivity basis functions toapproximate the spectral reflectivity are spectral reflectivitycomponents obtained by statistical analysis of the spectral reflectivitydata of a plurality of known object surfaces as a population; areintentionally extracted spectral reflectivity components; or are acombination of both.
 5. The light source estimation method according toclaim 2, wherein a vector space for said evaluation is a space in whichspectral distribution values for light reflected on an object surfacefrom a single virtual reference light source having a specific spectraldistribution are converted into a plurality of channels by fixedoperations.
 6. The light source estimation method according to claim 5,wherein a reference light source, said spectral distribution of which isfixed over a wavelength range, is used.
 7. The light source estimationmethod according to claim 2, wherein a plurality of light sources withdifferent known spectral distributions are taken to be said test lightsources; spectral distribution data for each test light source orcoefficients for computation corresponding to each test light source towhich said spectral distribution data is applied are stored in advance;and the data or coefficients are referenced at the time of said lightsource estimation.
 8. The light source estimation method according toclaim 2, wherein a plurality of different representative light sourcesare extracted and stored in advance as said test light sources fromamong spectral distribution data for various known light sources, fromamong coefficients to approximate the spectral distribution data byweighted linear sums of a plurality of light source basis functions, orfrom among indexes obtained using fixed computation formula from thespectral distribution data; and the spectral distribution data of eachof the test light sources, or the computation coefficients correspondingto each of the test light sources to which the data is applied, arereferenced at the time of said light source estimation.
 9. The lightsource estimation method according to claim 8, wherein, as theinformation for the plurality of different light sources stored inadvance, spectral distribution data for a specific light source orcomputation coefficients corresponding to a specific light source towhich the distribution data is applied are used; and said plurality oftest light sources are generated and referenced by appropriateselection, interpolation processing, or the like at the time of saidlight source estimation.
 10. The light source estimation methodaccording to claim 7, wherein said plurality of representative testlight sources are categorizable by the color temperature value of thelight source, by the physical light emission method of the light source,or by both.
 11. The light source estimation method according to claim 2,wherein, among the sensor response values of said image-pickup means,values with respect to all pixels or values with respect to pixelssampled at appropriate positions, in appropriate ranges, and atappropriate intervals within the spatial position of the image-pickupplane are used.
 12. The light source estimation method according toclaim 2, wherein, among the sensor response values of said image-pickupmeans, values with respect to only pixels the values for each channel ofwhich are in a specified range, or values with respect to all pixelsother than pixels the values for each channel of which are in aspecified range are used.
 13. The light source estimation methodaccording to claim 2, wherein, at the time of projecting sensor responsevalues of said image-pickup means into evaluation space, or prior tosaid time, scaling is performed at a fixed arbitrary ratio or at anappropriate ratio determined in advance according to image-pickupresults.
 14. The light source estimation method according to claim 2,wherein sensor response values of said image-pickup means are used afteradding noise, exposure error, or other temporally fluctuating quantitiessupposed in said image-pickup means, or after adding pixels to whichsuch fluctuating quantities have been added.
 15. The light sourceestimation method according to claim 2, wherein with respect to each ofsaid test light sources, a statistical quantity obtained from values ofsample pixels projected into evaluation space; a statistical quantityobtained from an image distribution indicating the frequencydistribution in evaluation space generated from sample pixels; astatistical quantity obtained from the image color gamut indicating theregion in the evaluation space in which sample pixels are distributed;or a combination of any two or more of these, are used, either withoutfurther modification, or after conversion into values by a fixedoperation, as said estimation criterion for an index of correctnessassumed in advance.
 16. The light source estimation method according toclaim 2, wherein a statistical quantity obtained from sample pixels insensor space of said sensor response values, or a statistical quantityobtained from values converted by a fixed operation from sample pixelvalues in sensor space, are projected into evaluation space with respectto each of said test light sources, and are used, either without furthermodification, or after conversion into values by a fixed operation, assaid estimation criterion for an index of correctness assumed inadvance.
 17. The light source estimation method according to claim 15,wherein, with respect to the spectral reflectivity of an object surface,an index of the correctness of each of said test light sources iscalculated in advance using statistical quantities added constraints orweighting formable in a specific region of the evaluation space, basedon the physical possibility in the range from 0 to 1 at each wavelengthand on an assumed probabilistic distribution in the real world in which,on average, there exist numerous surface approximating achromaticitywith a flat wavelength characteristic.
 18. The light source estimationmethod according to claim 2, wherein, with respect to each of said testlight sources, a correlation function of a reference color gamutrecorded in advance, referenceable, and indicating the range ofappearance in the evaluation space; with sample pixel values projectedinto the evaluation space; with the frequency distribution in theevaluation space generated from the sample pixels; with the region inthe evaluation space in which sample pixels are distributed; or with acombination of any two or more of same, is used as an index of saidestimation criterion.
 19. The light source estimation method accordingto claim 18, wherein a weighting distribution and region informationgenerated from the frequency distribution in the evaluation space ofvalues converted from spectral reflectivity data of various objectsurfaces into coefficients approximated by reflectivity basis functions,or of values obtained by converting said coefficients by a fixedoperation, are used as said reference color gamut.
 20. The light sourceestimation method according to claim 18, wherein a weightingdistribution and region information generated from the frequencydistribution of values, in which sensor response values which are eitherthe result of image pickup of a variety of actually existing scenes orthe result of predicting by numerical operations the images picked upfor a variety of virtual scenes are projected into evaluation space foreach scene using operations capable of calorimetrically approximatingfrom spectral sensitivity characteristics of said image-pickup means andfrom spectral distribution characteristics of the image-pickup lightsource measured at the time of image pickup of each scene, are used assaid reference color gamut.
 21. The light source estimation methodaccording to claim 18, wherein, with respect to spectral reflectivity ofan object surface, a weighting distribution and region informationgenerated from a frequency distribution determined based on a physicalpossibility in the range 0 to 1 at each wavelength and on an assumedprobabilistic distribution in the real world in which, on average, thereexist numerous surface approximating a chromaticity with a flatwavelength characteristic, are used as said reference color gamut. 22.The light source estimation method according to claim 19, wherein,before or after generating said reference color gamut from any of saidfrequency distributions or from a combination thereof, with respect tothe distribution in evaluation space, interpolation, extrapolation,removal, spatial filtering, or other processing according to fixedcriteria are performed.
 23. The light source estimation method accordingto claim 15, wherein, in generation of an index of correctness for eachof said test light sources, in order to emphasize the high colorsaturation region in which the difference between test light sourcesappears more prominently in the evaluation space, the image distributionis extracted and weighted operations are performed on the outline or inthe vicinity thereof in the image color gamut.
 24. The light sourceestimation method according to claim 15, wherein, with respect to theimage distribution or image region of sample pixels projected into theevaluation space, after performing interpolation, extrapolation,removal, spatial filtering, or other processing according to fixedcriteria, an index of correctness is calculated for each of said testlight sources.
 25. The light source estimation method according to claim15, wherein a plurality of different indexes generated from sample pixelvalues projected into a single evaluation space, or a plurality ofdifferent indexes generated from sample pixel values projected into aplurality of different evaluation spaces, are conjoined by numericmeans; are selected by conditional branching; or are both combined, togenerate a new index used to evaluate the correctness of each of saidtest light sources.
 26. The light source estimation method according toclaim 2, wherein the test light source having the highest index ofcorrectness among said plurality of test light sources is determined asthe estimated light source.
 27. The light source estimation methodaccording to claim 2, wherein the weighted averages of two or more lightsources having high correctness among said plurality of test lightsources is determined as the estimated light source.
 28. The lightsource estimation method according to claim 26, wherein a process, inwhich the light source with the highest index of correctness among saidplurality of test light sources is initially selected, and differentlight sources obtained in finely divided vicinity of said selected lightsource are referenced to generate indexes of correctness for each lightsource, is repeated.
 29. The light source estimation method according toclaim 26, wherein said test light sources include two or more categoriesaccording to physical light emission method; color temperature judgmentprocessing based on an index indicating that, within each category, thecolor temperature is closest to the color temperature of theimage-pickup light source, and light emission method judgment processingbased on an index indicating that the light source is closest to thephysical light emission method of the image-pickup light source, usingthe same or another index, are performed; and the estimated light sourceis determined from both the judgment results.
 30. The light sourceestimation method according to claim 26, wherein said test light sourcesinclude two or more categories according to physical light emissionmethod, and the estimated light source is determined based on an indexindicating a light source closest to the image-pickup light source, withrespect only to a test light sources belonging to a category specifiedby the user or to a category provided by category judgment meansdiffering from said estimation means.
 31. The light source estimationmethod according to claim 2, wherein the image-pickup light sourcedetermined by said estimation and a light source determined by anestimation method different from said estimation are conjoined bynumeric means, are selected by conditional branching, or are bothcombined, to determine the final estimated light source.
 32. Animage-pickup apparatus in which from sensor response values obtainedupon pickup of an image of an unspecified arbitrary object, image-pickupmeans having a plurality of different spectral sensitivitycharacteristics estimates spectral characteristics indicating color ofan unknown image-pickup light source irradiating an object, and whichuses, in color balance processing of the sensor response of saidimage-pickup means, the spectral characteristics which are the color ofthe estimated light source or parameters appropriate thereto,comprising: storage means for storing, for each test light source,parameters for projecting said sensor response values into an evaluationspace not dependent on said image-pickup light source by performingoperations which can be calorimetrically approximated from a pluralityof different said known spectral sensitivity characteristics of saidimage-pickup means and from the spectral characteristics of a pluralityof test light sources assumed in advance; projection conversion meansfor projecting said sensor response values into said evaluation spacenot dependent on the image-pickup light source using parameters storedin said storage means; evaluation means for evaluating the correctnessof said plurality of test light sources based on the image distributionstate of sample values of an image scene projected by said projectionconversion means; light source estimation means for estimating the finalimage-pickup light source to be determined as the estimated light sourceby conjoining in numerical formulas, by selecting through conditionalbranching, or by combining both of, an image-pickup light sourcedetermined by said estimation and a light source determined by anestimation method different from said estimation; and color balanceadjustment means which uses spectral characteristics, which are thecolor of the estimated image-pickup light source, or parametersappropriate thereto in color balance processing of the sensor responseof said image-pickup means.
 33. An image processing method in which fromsensor response values obtained upon pickup of an image of anunspecified arbitrary object, image-pickup means having a plurality ofdifferent spectral sensitivity characteristics estimates spectralcharacteristics indicating color of an unknown image-pickup light sourceirradiating an object; and which uses the spectral characteristics,which are the color of the estimated light source, or parametersappropriate thereto in color balance processing of the sensor responseof said image-pickup means, comprising the steps of: projecting saidsensor response values into an evaluation space not dependent on theimage-pickup light source through operations which can becalorimetrically approximated from spectral sensitivity characteristicsof known image-pickup means and from spectral characteristics of anassumed test light source; estimating the correct image-pickup lightsource by evaluating the correctness of a plurality of said test lightsources based on a state of distribution of sampled values of theprojected scene; estimating the final image-pickup light source to bedetermined as the estimated light source by conjoining using numericmeans, by selecting using conditional branching, or by both combined,the image-pickup light source determined by said estimation, and a lightsource determined by an estimation method different from saidestimation; and using, in color balance processing of the sensorresponse of said image-pickup means, spectral characteristics which arethe color of the estimated image-pickup light source, or parametersappropriate thereto.